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RESEARCH REPORT 

Anticipating College Enrollment: Adapting 
SuccessNavigator® for High School Students 

Samuel H. Rikoon, Travis Liebtag, Margarita Olivera-Aguilar, Jonathan Steinberg, & Steven B. Robbins 

Educational Testing Service, Princeton, NJ 

In this report, we describe the development of an extension of the SuccessNavigator® assessment for late high school settings. We 
discuss the assessment’s conceptualization and support its application with psychometric studies detailing scale development in terms of 
structural analyses, reliability, and several other aspects of validity. High School SuccessNavigator (HSSN) is an assessment of students’ 
noncognitive and academic skills important to their preparation for transitioning to higher education. The assessment provides a holistic 
review of students’ strengths and potential areas for improvement, and it was designed for use by counselors and administrators to aid 
their work in engaging students to facilitate skill development. The primary aim of HSSN is to provide a standardized metric by which 
to direct individualized support to students as they ready themselves for postsecondary enrollment. 

Keywords Noncognitive skills; holistic assessment; high school; college transition; academic skills; college guidance 
doi: 10.1002/ets2.12084 


Prior research has demonstrated positive relationships between the expression of noncognitive or psychosocial skills and 
both academic success and persistence in higher education (e.g., Porchea, Allen, Robbins, & Phelps, 2010; Poropat, 2009; 
Robbins, Allen, Casillas, Peterson, & Le, 2006; Robbins et al., 2004; Trapmann, Hell, Hirn, & Schuler, 2007). Specifically, 
the SuccessNavigator® assessment, created by Educational Testing Service (ETS), has also been shown to predict collegiate 
academic success (Markle, Olivera-Aguilar, Jackson, Noeth, & Robbins, 2013; Rikoon, Liebtag, Olivera-Aguilar, Robbins, 
& Jackson, 2014). While this body of research focuses on postsecondary outcomes, it does not diminish the importance of 
developing strong behavioral skills earlier in one’s academic career. In particular, the demonstration of strong noncognitive 
skills and appropriate academic behaviors prior to enrolling in a postsecondary institution is vital for long-term success 
beyond the high school level (Farrington et al., 2012). 

Venezia and Jaeger (2013) highlighted the importance of holistic approaches to high school intervention, suggesting 
that the most effective programs designed to encourage college readiness are able to identify and target specific areas of 
need. Furthermore, Lleras (2008) found that the exhibition of specific personality and noncognitive factors (i.e., conscien¬ 
tiousness, motivation, and cooperativeness) among high school students related to higher levels of educational attainment. 
High School SuccessNavigator (HSSN) was developed as an extension of the original SuccessNavigator assessment (Markle 
et al., 2013), which was focused on students who have already completed the transition to college. The broad goal of HSSN 
is to facilitate the standardized identification of noncognitive strengths and potential vulnerabilities among late-high 
school (11th and 12th grade) students. 

In this report, we detail the conceptual and methodological development of HSSN along several lines. We first discuss 
contextual issues motivating the expansion of the SuccessNavigator assessment to high school environments, proceeding 
to detail its multiple intended uses and utility to the field. Second, we provide a discussion of the HSSN scale development 
process and national field trial. This material is inclusive of item selection, structural analyses, scale reliability, validity evi¬ 
dence, and evaluating the assessment’s fairness across salient demographic groups. We conclude by noting the limitations 
of the current study and highlighting future directions for research. 

Counseling Burden in U.S. Public Schools 

The National Association for College Admission Counseling (NACAC) estimates that the average national student to 
counselor ratio in U.S. public K-12 schools is 478:1 (McDonough, 2006) 1 compared to the 250:1 ratio recommended by 
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the American School Counselor Association (ASCA) and the 100:1 ratio recommended by NACAC (American School 
Counselor Association, 2011, 2012; Clinedinst & Hawkins, 2011). Further complicating the issue of high school student 
to counselor ratios is that, although ASCA provides recommendations defining the role and responsibilities of profession¬ 
ally certified counselors, 2 those of a typical practicing counselor are difficult to determine and must (by virtue of differing 
access to resources) vary widely between schools and districts. Examples of duties assigned to counselors beyond the tra¬ 
ditional tasks of course planning and managing behavioral or disciplinary issues include (a) fulfillment of requests for 
student data, (b) standardized test coordination and administration, (c) leading partnerships with community organiza¬ 
tions, (d) development and implementation of intervention strategies, (e) working intensively with students deemed at 
risk of academic failure, (f) managing budgetary restrictions, and (g) precollege advisement (Burnham & Jackson, 2000; 
Perna et al., 2008; Woods & Domina, 2014). This is not to suggest that any of the preceding activities are inappropriate for 
counselors to engage in, only that the scope of their responsibilities has the potential to become burdensome even before 
considering any time spent interacting with students. 

With specific attention to college advisement, McDonough (2006) estimated that “students in public school can expect 
less than an [one] hour of postsecondary education counseling during the entire school year” (p. 3). Research by Eng- 
berg and Gilbert (2014) showed that the amount of time high school counselors spend on college-related activities and 
advisement is related positively to college attendance rates. Counselor availability, their level of student interaction, and 
the extent to which schools offer institutional support and college planning resources are all key contributors to the like¬ 
lihood of a high school student pursuing higher education (Belasco, 2013; Engberg & Gilbert, 2014; Perna et al., 2008; 
Woods & Domina, 2014). 

The impact of high school counselors on the pursuit of higher education appears to be particularly important among 
students from low-income backgrounds. Belasco (2013) found that students with a lower socioeconomic status (SES) 
“may have yielded more benefit from their relationship with a school counselor” (p. 795; emphasis added) compared to 
students with a higher SES. Perhaps not surprisingly, given the positive relationship between educational attainment and 
income levels (e.g., Bailey & Dynarski, 2011), lower individual and institutional SES have also been associated with a lower 
likelihood of pursuit, enrollment in, and completion of a postsecondary program (Belasco, 2013; Oseguera, 2013). This 
finding is perhaps a direct result of higher student to counselor ratios and increased burdens of responsibility in schools 
primarily serving students in low-income situations, relative to schools located in affluent areas (Engberg & Gilbert, 2014). 

In addition to the potential institutional obstacles faced by students in low-income situations, they also experience a 
significant degree of academic “undermatch.” An undermatch occurs when student’s academic credentials would have 
gained that student access to a college or university more selective than the postsecondary institution at which the student 
actually enrolled (Smith, Pender, & Howell, 2013). Research over the past decade has found that students in low-income 
situations are especially prone to undermatching (Avery et al., 2006; Dillon & Smith, 2013; Griffith & Rothstein, 2009; 
Hoxby & Avery, 2013). Most often, undermatching is not the direct result of a higher education admissions process but 
is derived from students not submitting an application to a more selective institution in the first place (Dillon & Smith, 
2013). Prior studies have revealed several factors contributing to student undermatch, including a lack of information 
about application and admission processes, concerns about the affordability of selective colleges, and greater distance 
between a student’s home and selective versus less selective institutions (Avery et al., 2006; Griffith & Rothstein, 2009; 
Hoxby & Avery, 2013; Roderick, Nagaoka, Coca, & Moeller, 2008). 

Utility of High School SuccessNavigator 

The combined importance of assisting high school students reach their full educational potential and alleviating the cur¬ 
rent pressure on overburdened counselors to serve all of their students effectively (e.g., by working to prevent undermatch) 
has motivated nationwide initiatives to support students in low-income situations or those otherwise in need. 3 As these 
types of programs, organizations, and high school counselors in general strive to achieve their respective missions, they 
naturally imply (if not explicitly claim to deliver) efforts to cultivate important noncognitive skills among the students 
they serve. For example, a student without prior plans to apply to competitive colleges or universities might be required 
to develop increased levels of perseverance, organization, and stress management to complete the process successfully. 

It is in these and similar contexts where an assessment such as HSSN can potentially meet an essential need by enabling 
student support staff to gauge each student’s areas of greatest need and focus their efforts accordingly. Furthermore, the 
counseling process may become more efficient through the implementation of an assessment that students complete on 
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their own, thereby freeing up their already limited time in direct contact with a counselor for the actual delivery of 
guidance. Recognizing that counselors will naturally form a casual impression of student noncognitive skill levels and 
behavioral characteristics (e.g., through direct interactions or others’ reports), the development of HSSN provides a more 
rigorous metric on which these qualities in a supportive, low-stakes context can be evaluated. 

HSSN is an assessment carrying low or no stakes. While assessments that help in making high-stake decisions are 
associated with the offer or denial of a specific outcome or opportunity for test takers dependent on their performance 
(e.g., graduation from high school, admission to a particular college, advanced course placement, employment offer), a 
low-stake classification implies no such direct impact of assessment results on the individual level. The primary intent 
underlying the development of HSSN was to provide an advising tool demonstrating psychometric rigor to facilitate 
individualized student counseling. In practice, HSSN should provide a common language, series of metrics, and support¬ 
ive materials to aid counselors and students alike in solidifying the skills and behaviors necessary for success in higher 
education. More specifically, an implementation of HSSN could serve four primary purposes: 

1. Assess student strengths and areas of need on a broad spectrum of noncognitive skills and behaviors identified as 
significant contributors to academic success on the postsecondary level. 

2. Provide usable feedback to high school guidance counselors and college advisors in the form of HSSN score reports 
and associated resource guide materials. Counselors could make use of such feedback to coordinate the individu¬ 
alized delivery of institutional resources and interventions to support student needs. 

3. Provide specific details regarding students’ stated intentions toward higher education. Combining these data with 
noncognitive skill levels would help gauge students’ motivation to pursue postsecondary opportunities. In addition 
to post-high school plans, HSSN reports may also include related background data on each student’s grade point 
average (GPA) and standardized test scores (if available). 

4. Aggregate and report student data at the school level. Administrators could use aggregate reports to determine the 
relative strengths and weaknesses of their specific student body as well as where their institution lies in comparison 
to the large sample of students who participated in the HSSN field trial (described in detail in the next section). 
Similar to their use on the college level, high school leaders could employ institutional reports to inform program 
development catered to the specific needs of their student population. 

Development of High School SuccessNavigator 

HSSN encompasses four general construct domains (see Table 1 for a summary of all targeted constructs). These are aca¬ 
demic skills, achievement motivation, self-management, and social support. With some modification (see the next section 
for detail), these domains are similar in scope to those assessed by the original college-level SuccessNavigator assessment 
(Markle et al., 2013), which was theoretically anchored to the five-factor model of personality (also known as the Big 5; 
McCrae & Costa, 1987). HSSN, being an adaptation of this assessment, incorporates focal constructs that remain primarily 
aligned in theory with three of the Big 5 personality constructs: conscientiousness, emotional stability, and extraversion. 

Previous research has shown higher levels of conscientiousness to be positively associated with academic success at all 
levels of education through to adulthood (Ackerman & Heggestad, 1997; de Fruyt & Mervielde, 1996; Noffle & Robins, 
2007; O’Connor & Paunonen, 2007; Shiner, Masten, & Roberts, 2003). Typical descriptions of the construct include char¬ 
acteristics such as goal striving, drive, effort, and an ability to organize and follow through with plans. These and similar 
elements are conceptually related to the academic skills and achievement motivation domains included in HSSN. Aca¬ 
demic skills include individuals’ organizational capabilities (e.g., managing schoolwork and time), their ability to meet 
classroom expectations (e.g., behaviors directed toward meeting course requirements), and the extent to which they per¬ 
sist in their work and tasks despite any difficulties encountered. Achievement motivation centers on the extent to which 
students maintain a focus on pursuing higher education (e.g., perceiving college as valuable to their future) and believe 
in their own ability to succeed or achieve their academic goals (i.e., academic self-efficacy; Bong, 2001; Bong & Skaalvik, 
2003; Chemers, Hu, & Garcia, 2001). 

Emotional stability (interpreted as exhibiting low levels of the Big 5 dimension neuroticism; i.e., being even-tempered, 
maintaining calm when faced with stress) has been found in previous meta-analytical research to be related to academic 
achievement and persistence (Poropat, 2009; Richardson, Abraham, & Bond, 2012; Robbins et al., 2004). Indicators of 
emotional stability (such as the extent to which students are able to control their responses to external stress and anxiety) 
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Table 1 Construct Map of High School SuccessNavigator Scales 


Domains and Subskills 


Sample Assessment Item 


Academic Skills: Tools and strategies for academic success 

Meeting Class Expectations: Doing what’s expected to meet the 
requirements of your course including assignments and 
in-class behavior 

Organization: Strategies for organizing work and time 

Perseverance: Determination and effort to pursue tasks/goals 
despite difficulty or delay 

Achievement Motivation: Active pursuit of personal and academic goals 

Commitment to College Degree Goals: Perceived value and 
determination to succeed and complete college 

Academic Self-Efficacy: Belief in one’s ability to perform and 
achieve in an academic setting 


• I come to class prepared to take tests. 


• I follow a set routine. 

• I stay focused on tasks until they are done. 


• I am making plans to go to college. 

• I am confident that I will succeed in school this 
year. 


Self-Management: Internal reactions and the belief in a personal ability to succeed 

Managing Test Anxiety: General reactions to test-taking • I feel nervous when I take a test, 

experiences, including negative thoughts and feelings 
(e.g., worry, dread) 

Managing Stress: Tendency to feel frustrated, discouraged, or • I get worried when things do not go as planned, 

upset when under pressure or burdened by demands 


Social Support: Connecting with people and student resources for success 

Connectedness: A general sense of belonging and engagement 
School-related Help Seeking: Attitudes about and tendency to 
seek help from established resources 


• I feel like a part of my community. 

• I feel like there are people at my school I can talk 
to when I need help. 


Lacking Barriers: Financial pressures, family responsibilities, 
conflicting work schedules, and limited institutional knowledge 


• Family or other outside responsibilities interfere 
with my schooling. 


are assessed in HSSN’s self-management domain. Self-management is defined by students’ responses to and their ability 
to manage general personal stressors as well as the severity of their reactions to the experience of test taking (Chapell et al., 
2005). Test anxiety represents a specific and important challenge to negotiate (if not overcome) for high school students 
preparing to apply to colleges or universities (Hill, 1984; Zeidner, 1998). This issue is relevant for students preparing to 
apply to postsecondary institutions where their performance on a standardized high-stakes assessment (e.g., SAT®, ACT) 
is likely to serve as an initial threshold for admission as well as those who may be required to take academic placement 
exams (carrying similarly high stakes for course placement) upon enrolling in less selective institutions (Charlesworth, 
Fleege, & Weitman, 1994; Cizek & Burg, 2006; Segool, Nathaniel, Mata, & Gallant, 2014). 

Extraversion is a personality trait characterized by an individual’s aptitude and willingness to engage and connect with 
other people. Meta-analyses have demonstrated mixed results in terms of relationships between extraverted attitudes or 
behaviors (e.g., social involvement) and success in higher education, generally showing them to be either near-zero or 
moderately positive (Poropat, 2009; Richardson et al., 2012; Robbins et al., 2004; Trapmann et al., 2007). HSSN addresses 
this area indirectly through its assessment of the social support mechanisms students may have access to as they work to 
complete high school. Attitudes and behaviors used to define social support in HSSN include the extent to which students 
feel connected with their peers and community at large as well as students’ knowledge of and comfort with utilizing the 
institutional resources offered by their school (e.g., academic support or other counseling). 

Apart from the four primary domains assessed by HSSN, the instrument also includes several items designed to assess 
the extent to which students experience external barriers to their academic success (e.g., family issues, other extracurricu¬ 
lar responsibilities, illness). These items address specific challenges to students’ time and attention, by definition exerting 
a negative influence on their ability to complete the required schoolwork and focus on furthering their education. All else 
held constant, experiencing fewer such barriers would be expected to relate positively to academic achievement. 

In the remainder of this report, we describe the item development process, a large-scale national field trial, and psy¬ 
chometric studies to evaluate the dimensionality, reliability, and validity of HSSN. 
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Method 

Item Generation 

The 93 Likert-type self-report item statements included in the original SuccessNavigator assessment (Markle et al., 2013) 
formed the initial basis for the HSSN item pool. Doctoral- and master’s-level researchers reviewed all the items of the 
SuccessNavigator assessment in detail for appropriateness of language (e.g., sentence complexity, estimated reading level, 
specificity to a college setting), alignment with previously established subskill definitions, and issues of potential cultural 
sensitivity. Once independent evaluations of the items were completed, the same researchers collaborated to review and 
compare recommendations for suggested edits to the item pool for a high school audience. This collaborative review also 
included the development of a smaller pool of new candidate items to both replace those removed during the initial review 
and add items targeting specific subskills in cases where fewer than nine remained. All items were accompanied by the 
same four-category response option format asking respondents to rate the relative frequency with which they experienced 
or engaged in a given attitude or behavior, respectively. The category labels used as response options were never or rarely, 
sometimes, often, and usually or always. 

This process resulted in a preliminary pool of 104 HSSN items among which the most significant revision from the 
SuccessNavigator assessment’s 93-item college pool was the removal of an 8-item subset intended to target institutional 
commitment (due to the concept’s general lack of applicability to a high school population). In place of this subset, 11 new 
statements were introduced with the intention of targeting a student’s level of perseverance. Perseverance (also known as 
persistence) was considered a facet of conscientiousness and defined in HSSN as a student’s level of determination and 
effort to pursue tasks or goals despite difficulty or delay (MacCann, Duckworth, & Roberts, 2009; Roberts, Lejuez, Krueger, 
Richards, & Hill, 2014). 

Together with the full 104-item candidate pool, narrative descriptions of all 10 focal HSSN subskills were submitted 
to a doctoral-level psychometrician and a second research associate for their independent review. Their review focused 
on the same issues as described previously. This second review process resulted in the removal of four items from the 
overall pool. The remaining and finalized set of 100 items included 18 statements with language identical to the original 
SuccessNavigator assessment, 38 new statements, and 44 statements containing language slightly modified from the items 
in the original SuccessNavigator assessment. An example of the latter was this revision: “I feel like part of campus life” to 
“I feel like part of my community.” 

In addition to the candidate pool of assessment items, HSSN included 26 background information questions to establish 
a thorough description of each respondent’s demographic characteristics (e.g., age, gender), academic background (e.g., 
standardized test scores, GPA), and academic aspirations (e.g., anticipated level of degree attainment, intended major if 
known). In order to ensure that HSSN items met ETS standards for quality and fairness, an ETS fairness and sensitivity 
review panel examined all assessment and background informational items. Reviewers found that all items complied with 
ETS fairness guidelines and approved all HSSN items for use in the field without further revision. 

National Field Trial 
School Sample 

Twenty-three public high schools across the United States participated in the national field trial of HSSN between February 
and June 2014. Initial contact with schools was in the form of an e-mail invitation to participate in the study. Invitations 
were sent to a master list of more than 6,000 high schools compiled through a merge of school contact information 
originating from existing ETS institutional resources, members of the National Community College Advisory Council, 
and a school recruitment database maintained by ETS’s data collection services division. At the school level, there were no 
specific requirements for eligibility to participate in the field trial. Schools were included in the sample on a first come, first 
served basis as ETS received responses to the initial invitation. Each school’s primary point of contact was responsible for 
remaining in regular contact with ETS Data Collection Services or research staff to arrange study logistics and coordinate 
the administration of HSSN at the school’s location. 

Sixteen participating public, noncharter schools and one charter school received an incentive for participation equal to 
$5 for each student who successfully completed the assessment. Six charter schools that participated each received a flat 
incentive of $500. Individual students received no incentive for their participation in the study, which in all cases took place 
during the course of their normal school day and solely involved completing the assessment within a single class period. 
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Table 2 Field Trial Student Responses by School Type, Region, and School 



Frequency 

Percentage 

Charter status (N = 3,187) 

Charter 

889 

27.9 

Noncharter 

2,298 

72.1 

Region {N = 3,187) 

Midwest 

347 

10.9 

Northeast 

1,209 

37.9 

South 

991 

31.1 

West 

640 

20.1 

Schools (N= 3,187) 

1 

187 

5.9 

2 

142 

4.5 

3 

186 

5.8 

4 

17 

0.5 

5 

25 

0.8 

6 a 

26 

0.8 

7 

345 

10.8 

8 

33 

1.0 

9 

426 

13.4 

10 

22 

0.7 

ll a 

326 

10.2 

12 a 

60 

1.9 

13 

91 

2.9 

14 

145 

4.5 

15 

51 

1.6 

16 

136 

4.3 

17 

55 

1.7 

18 

166 

5.2 

19 

271 

8.5 

20 a 

179 

5.6 

21 a 

180 

5.6 

22 a 

30 

0.9 

23 a 

88 

2.8 


“Charter school. 


Table 2 provides a summary of the number of students participating in the field trial from each school as well as the 
geographic distribution of students in the sample. Participating institutions were located across the United States in the 
southern (N= 8), midwestern (N = 6), western (N = 5), and northeastern (N = 4) regions of the country. Institutions also 
served a socioeconomically diverse population of students, with 57% of their students on average eligible for a free (47%) 
or reduced-price (10%) lunch program (range 4-88%). In 16 of the 23 field trail schools, 50% or more of the student body 
was eligible for such a program. The preceding information comports approximately with national data indicating that 
51% of all public school students met similar eligibility criteria during the 2012-2013 school year (U.S. Department of 
Education, National Center for Education Statistics, Common Core of Data, 2014). 


Student Sample 

The population of interest for a national field trial study of HSSN was students preparing for the transition to higher 
education. As a result, student eligibility within participating high schools was limited to 11th and 12th grade students. 
Table 3 provides a demographic description of the student sample. To summarize, the sample was split evenly by gender, 
with a slight majority (56%) in the 11th grade versus 12th. Students participating in the HSSN field trial were ethnically 
diverse, with the largest subgroup of students (45%) identifying as White, 21% identifying as belonging to more than one 
racial or ethnic group, 17% as exclusively Hispanic, and 13% exclusively as Black. Of the sample, 67% were native English 
speakers. 

Contextual information was collected as part of the HSSN background questionnaire related to both challenges faced 
by students and any services they had availed themselves of while at school. Table 4 displays a summary of the types 
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Table 3 Field Trial Responses by Student Demographics 




Frequency 


Percentage 

Gender (N = 3,025) 





Male 


1,502 


49.7 

Female 

Race (N = 3,046) 


1,523 


50.4 

Black 


409 


13.4 

White 


1,384 


45.5 

Hispanic 


521 


17.1 

Multiracial 


625 


20.5 

AI/AN 


22 


0.7 

Asian 


75 


2.5 

NH/PI 

Grade (N = 3,135) 


9 


0.3 

11 


1,747 


55.7 

12 


1,388 


44.3 

English as Primary Language (N = 3,060) 





Yes 


2,330 


76.1 

No 


730 


23.9 

Note. AI/AN = American Indian or Alaskan Native; NH/PI 

= Native Hawaiian or Pacific Islander. 


Table 4 Percentage of Students Experiencing Impact of External Barriers During the Past 12 Months 



No impact 

Little impact 

Some impact 

Significant impact 

Family obligations (N = 3,010) 

48.8 

26.2 

17.0 

8.0 

Financial difficulties (N= 3,015) 

34.0 

30.2 

24.7 

11.1 

Issues with health (N = 3,018) 

56.7 

24.5 

13.5 

5.4 

Legal issues (N = 2,988) 

75.4 

14.7 

6.8 

3.2 

Personal problems (N = 3,020) 

16.1 

27.3 

32.8 

23.8 


Table 5 Number and Percentage of Students Using Institutional Services in the Past Year 


Frequency 

Percentage 

Advising (N= 2,172) 

446 

20.5 

Counseling (TV = 2,174) 

494 

22.7 

Intramural sports (N = 2,369) 

923 

39.0 

Tutoring (N = 2,301) 

719 

31.3 

None used (N = 2,291) 

1,277 

55.7 


of challenges students in the field trial sample reported having faced in their daily lives over the previous year. Table 5 
reports the types of school-based support and activities students had taken advantage of during their high school careers 
to date. Across all areas (i.e., accounting for students who selected more than one response to the questions summarized 
in Table 4), 65% of students completing that portion of the background information form reported having experienced 
either some or significant impact on their life as a result of family, financial, health, legal, or personal problems. Of these, 
50% (32% of the overall sample) characterized the impact(s) as significant. Turning to school-based support (summarizing 
across all areas depicted in Table 5), only 36% of the sample reported having utilized any counseling, advising, or tutoring 
services at some point during high school (i.e., 36% of the sample selected one or more of the services). 

Study Procedures 
Data Collection 

Schools were given the option of using either online or paper forms to administer the HSSN assessment to students. The 
order of the HSSN item presentation was randomized both in the online form and on the three paper versions generated. In 
schools utilizing paper forms, the forms themselves were distributed to students in random order. Students were instructed 
that the assessment was designed to measure their attitudes and behaviors, that there were no right or wrong answers to the 
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assessment, and that there would be no time limit. The assessment’s instructions also emphasized that none of a students’ 
responses would be shared with their teachers or school. 

ETS maintains an institutional review board that approved all data collection procedures and materials described 
herein. No individually identifying student information was collected during the course of the HSSN field trial. Instead, a 
unique identification number was used to match student HSSN assessment records to any institutionally reported student 
outcome or other individual-level data. 

External Evidence for Convergent and Discriminant Validity 

In addition to the HSSN assessment itself and background information questions, students were also administered a 50- 
item personality assessment targeting the Big 5 traits of agreeableness, conscientiousness, emotional stability, extraversion, 
and intellect (or imagination). This assessment contained items adapted from the open-source International Personality 
Item Pool (IPIP; Goldberg et al.,2006). The purpose of including abroad-domain personality assessment in the HSSN field 
trial was to incorporate a concurrent but independent method of assessing evidence for the convergent and discriminant 
validity of each of the targeted HSSN constructs. Item presentation order in the IPIP assessment was randomized using 
the same strategy used for the HSSN items, with the block of 50 randomized IPIP items always appearing immediately 
following the block of 100 HSSN items. It is important to note here that students saw no distinction between the two 
forms during the course of the assessment (i.e., it appeared that they were completing a single 150-item survey). The only 
difference between the HSSN and IPIP items was that in the case of the IPIP, all items carried a four-category response 
option format asking respondents to rate their level of agreement with each statement versus HSSN item response options 
soliciting ratings of their relative frequency of engaging in a given behavior or attitude. The category labels used as response 
options for the IPIP items were strongly disagree, disagree, agree, and strongly agree. 

Student Outcomes 

Included within each school’s memorandum of understanding about participation in the HSSN field trial was an agree¬ 
ment to provide ETS with both student behavioral and academic outcome data as they became available at the close of 
the 2013-2014 school year. Although practical difficulties prevented all schools from reporting all requested elements, 
at least nine schools (minimum student N = 1,092) reported each of the following variables to enable the estimation of 
external validity relationships between HSSN scales and student academic or behavioral outcomes. Academic outcome 
variables collected were standardized test scores (e.g., ACT, SAT), cumulative high school GPA (HSGPA), and current 
semester GPA; student behavioral outcomes collected were the number of disciplinary infractions reported and their rate 
of attendance during the current academic year (2013-2014). 

Data Preparation 

A professional data services firm was contracted to convert all paper-form student response data to a digital database. 
This process involved duplicate entry and independent verification of data accuracy by two staff people at the firm. We 
standardized and merged all school-reported student outcome data with the previously merged noncognitive assessment 
and background information data. In all the cases where both were available, institutionally reported data were given 
priority over student-reported data. The process followed a similar procedure to that used to concord similar data received 
from multiple institutions in the field trial of the original SuccessNavigator assessment (Markle et al., 2013). To summarize, 
all GPA data were converted to a standard scale by schools (range 0.0-4.0). Standardized test scores were concorded 
to the ACT scale (range 1-36) using available conversion tables (ACT, 2010; The College Board, 2009). Data received 
on behavioral infractions (e.g., misconduct, detention, suspension) were tallied to indicate a total number of incidents 
recorded for each student, and attendance data were scaled to indicate the percentage of days in a standard 180-day school 
year during which the student was in school. 

It is important to note several caveats to the interpretation of the student outcome variables discussed previously. 
First, data were not available to characterize the severity of behavioral infractions coded by each school, limiting the 
interpretation of this variable to a simple count of the number of times a student was referred to school administrators 
for any disciplinary reason. Second, HSGPA is naturally imperfect as a standard measure of academic achievement given 
the myriad contextual factors underlying its construction. These include, for example, a variation in academic standards 
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between schools (Adelman, 1999, 2006) and the extent to which teachers weigh different criteria in their assignment of 
course grades (J. D. Allen, 2005; Brookhart, 1993; McMillan & Nash, 2000). Understanding its limitations, HSGPA in the 
current study was interpreted as a general indication of the level of students’ performance in their coursework relative to 
others within their high school environment. A similar interpretation was applied to the concordance of SAT and ACT 
scores to a common scale on which merged scores were taken to indicate students’ performance relative to the national 
population of those taking the same tests (Dorans, 2004). As described in detail in the following pages, multilevel modeling 
was used to account for systematic between-school differences in all analyses of student outcome variables. 

Completion of the data merging resulted in a master HSSN field trial dataset containing 3,705 observations. Initial 
descriptive analyses revealed a subset of observations (N = 518) that were culled from the master HSSN dataset as invalid 
responses. An invalid designation was assigned to observations meeting one or more of the following criteria: 

• The observation was generated by a school administrator testing the online assessment (N = 46). 

• The observation was missing more than 40% of HSSN item responses (N = 163). 

• The student’s home school could not be identified (N = 297). 

• The observation exhibited an invariant response pattern (N = 47). Invariance in this context was indicated by 95 or 
more of the 100 possible HSSN item responses being in the same response category. This was taken to indicate that 
a student simply filled in identical responses for all items without considering them individually. 

These conditions resulted in a final HSSN dataset of 3,178 valid observations available for further analysis. These obser¬ 
vations made up the source dataset for all psychometric and other analyses reported herein. Finally, with regard to data 
preparation, ETS research staff reverse coded all negatively worded HSSN and IPIP items (e.g., “I get worried when things 
do not go as planned.”) so that higher scores on all the items intended to target a given scale indicated higher levels of a 
desirable noncognitive skill or behavior (e.g., managing stress). 

The final HSSN dataset was randomly split into two halves to facilitate modeling the assessment’s factor structure. 
These halves were referred to below as the index (N = 1,593) and reserve (N = 1,594) subsamples, respectively, with new 
random subsamples created for testing each of the hypothesized HSSN domains. The index subsample for each domain 
was used to fit initially hypothesized factor structures for each HSSN content domain and to test any model modifications 
(e.g., dropping a poorly performing item from the analysis) suggested by a review of initial results from both theoretical 
and empirical perspectives. The reserve subsample was used solely to test the final measurement model for each HSSN 
domain using an independent dataset. Full modeling procedures are described in detail below (see “Factor Analysis”). 

Multilevel Structure 

Students in the HSSN field trial sample were naturally nested within schools, creating potential issues of observational 
dependency between students within the same school. In other words, on a given assessment, students within the same 
school may behave more similarly to one another versus students from different schools simply as a function of the former 
having attended the same school. Multilevel modeling (Raudenbush & Bryk, 2002) was conducted to obtain an impression 
of the extent to which student clustering within schools (i.e., hierarchically structured data) might account for item-level 
variance and thus influence the results of any single-level analysis. A series of unconditional, multilevel models was esti¬ 
mated to partition item response variance into its student- and school-level components. These variance components 
were used to estimate the intraclass correlation (ICC) for each HSSN and IPIP assessment variable as well as all stu¬ 
dent outcome variables. The ICC indicated the proportion of total item response variance attributable to between-school 
differences, with the remainder observed between students within schools. Where item-level ICCs were below .05, fur¬ 
ther multilevel analysis was considered unnecessary due to the nominal amount of cluster-level variance present (Brown, 
2015). A more lenient criterion was suggested by Kline (2011), who indicated that multilevel modeling should be used 
when a variable’s ICC lies above 0.10. 

Dimensionality 
Factor Analysis 

It is important to recognize that HSSN item response data are represented as ordered categories. A well-developed litera¬ 
ture exists that details the negative consequences (e.g., spurious dimensionality) inherent to the treatment of ordinal item 
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response data (represented as only a few discrete categories) as continuous data in factor analytical studies (Bernstein & 
Teng, 1989; Finney & DiStefano, 2006; Flora & Curran, 2004; Garrido, Abad, & Ponsoda, 2011; Mislevy, 1986). Following 
recommendations in the literature (Beauducel & Herzberg, 2006; Flora & Curran, 2004), in the present study, we used 
matrices of polychoric correlations, which were estimated via robust weighted least squares (WLSMV) as implemented in 
M plus 7.3 (Muthen & Muthen, 2012), as input for all reported measurement models. As is standard practice when using 
this estimator, missing data in the preceding models were handled via pairwise deletion. All nested model comparisons 
were conducted using the DIFFTEST option in Mplus. 

Initial unconditional multilevel analyses of each of the 100 potential HSSN indicators revealed that only nominal 
amounts of between-school variance were present in the data. On average, 2.29% (SD= 1.46%) of item response vari¬ 
ance was accounted for by school membership (range 0.13-9.01%). Because 96 of the 100 items exhibited ICCs below 
5% and none exceeded 10%, multilevel modeling was deemed unnecessary in the context of establishing the assessment’s 
dimensionality. Following Brown’s (2015) recommendation, all confirmatory factor analytical (CFA) models reported 
herein reflect model fit statistics and standard errors adjusted to account for whatever minor amount of clustering was 
present in the HSSN data. 4 

From its inception, the measurement structure of HSSN was designed to be multidimensional in nature. This is evi¬ 
denced by the varied scope of its targeted constructs and the item content generated to address them. Each individual 
item was written to target both a specific construct and a more broadly conceptualized domain. For example, the item “I 
feel connected to my peers” was hypothesized to be an indicator of students’ levels of attachment to their community but 
was also thought to provide some indication of the strength of their social support network. Another example is the item 
“I keep track of my school materials” that was hypothesized to speak directly to students’ use of specific organizational 
strategies as well as provide an indication of their academic skills in general. 

Modeling all HSSN items together within the same CFA (each specified to load on one of 10 hypothesized latent 
psychosocial factors) would result in a model estimating 445 free parameters (90 item loadings + 10 factor variances 
+ 300 item thresholds + 45 interfactor correlations). This model was considered too complex to estimate the given index 
and reserve subsample sizes of approximately 1,590 observations (i.e., there would only be a few observations per model 
parameter). In order to maintain the theoretical grouping of HSSN factors within larger, conceptually related domains 
(while also achieving sufficient model parsimony), we fit a separate correlated-traits model for each hypothesized HSSN 
domain to that domain’s index subsample. The goodness of fit of all models was evaluated through a review of multiple 
overall fit indices, with model parameter estimates (e.g., factor loadings) also reviewed as an indication of localized fit. 
Models were considered to have demonstrated a close fit to the data when they exhibited both a comparative fit index 
(CFI) nearing or above .95 and a root mean square error of approximation (RMSEA) nearing or below .06 with upper 
90% confidence limit also nearing or below .06 (Hu & Bentler, 1999). Given the recent critiques of the appropriateness of 
implementing such fixed rules for accepting versus rejecting a given model (Brown, 2015; Chen, Curran, Bollen, Kirby, 
& Paxton, 2008; Kline, 2011; Marsh, Hau, & Wen, 2004) and the extent to which simulations to develop them may lack 
generalizability to our specific models and ordinal item response data, the classic criteria specified here were understood 
to be only approximate guidelines. 

In terms of evaluating model parameter estimates, HSSN items were deemed to be functioning poorly within a model if 
they exhibited standardized factor loadings below .20. In cases where a low loading was observed for an item, the item was 
dropped from the analysis. Exceptions to these rules were made only in cases wherein an item’s content was considered to 
represent an element essential to maintaining the intended definition of a targeted construct. All model modifications were 
implemented one at a time, with the fit and parameter estimates of the modified model studied using the index subsample 
to determine whether any further modifications were required. Once a final model was determined for a given domain, the 
same specification was applied to the appropriate reserve subsample to confirm an acceptable model fit. Finally, the model 
was fit to the overall HSSN sample (N = 3,187). We report model fit statistics below for both subsamples and the overall 
dataset for all retained HSSN domains as well as parameter estimates from all models fit to the overall HSSN sample. 

Reliability 

Although Cronbach’s alpha is the most widely reported method of assessing reliability, its traditional estimation uses 
Pearson product-moment correlations that underestimate the magnitude of bivariate relationships when applied to ordi¬ 
nal data (see Garrido, Abad, & Ponsoda, 2013 for a recent discussion of this issue). As a result, Cronbach’s alpha as typically 
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computed is a biased indicator of reliability for scores derived from ordinal response variables. To address this issue, we 
estimated factor reliability using a version of Cronbach’s alpha intended for studies making use of polytomous data (a Q ; 
Gadermann, Guhn, & Zumbo, 2012; Zumbo, Gadermann, & Zeisser, 2007). This version of Cronbach’s alpha makes use of 
polychoric correlations to estimate the reliability of unobserved continuous variables assumed to underlie our observed 
HSSN item response data. 

Scoring 

In order to remain consistent with the collegiate version of the SuccessNavigator assessment, student scores for each 
HSSN dimension were computed as the mean response level across all retained items. 5 Scores were produced for each 
HSSN general domain (e.g., Academic Skills) and subskill (e.g., Organization, Meeting Class Expectations) as well as 
Lacking Barriers (reflecting a reversed coding of the extent to which students reported experiencing external conflicts 
with their schooling). It is important to note that any reported HSSN score is not intended for making fine-grained dis¬ 
tinctions between individual students (Bollen, 1989). HSSN was designed for use in low- or no-stakes contexts to facilitate 
the support and development of students preparing for the transition to higher education, and thus (similar to its prede¬ 
cessor) any operational assessment would report student score categories (e.g., by quartile) as opposed to specific point 
estimates. 

Validity Evidence 

Convergent and Discriminant Evidence for Validity 

An important part of establishing the overall validity of an instrument involves demonstrating that its components relate to 
both each other and external measures as expected (American Educational Research Association, American Psychological 
Association, & National Council on Measurement in Education, 2014). In this context, the convergent and discriminant 
evidence for the validity of HSSN scores is established to the extent that scales addressing theoretically similar constructs 
relate more strongly to one another (convergent evidence) than do scales targeting theoretically different constructs (dis¬ 
criminant evidence). We present convergent and discriminant evidence for the validity of HSSN scores in two ways. From 
an internal perspective, we review correlations between both HSSN domain and subskill scores. Turning to external evi¬ 
dence, we examine relationships between HSSN scores and the concurrently administered Big 5 personality assessment 
based on the IPIP. It is noted that (as was the case with regard to HSSN items) average ICCs for the IPIP items revealed 
only nominal amounts of between-school variance. On average, 1.20% (SD = 0.88%) of IPIP item response variance was 
accounted for by school membership (range 0.23-4.47%). Because none of the 50 IPIP items exhibited ICCs above 5% 
(i.e., only a trivial proportion of overall IPIP item variance was associated with attendance at a given school), Pearson 
product-moment correlations were used to assess the extent of relationships between HSSN and IPIP scores. 

Relationships With Student Outcomes 

Information on college enrollment status was not available for the current sample because students were still enrolled in 
high school during its collection. Such data are planned for future collection after 11th grade students in the current sample 
have matriculated. We were, however, able to gather a set of broad outcomes data summarizing both a student’s academic 
progress to date and level of compliance with school behavior and attendance policies. These outcomes were taken as 
initial indicators of student success to facilitate an exploratory study of the external predictive validity of HSSN scores. 
Correlations were thus estimated between HSSN scores and measures of student academic achievement (standardized 
test scores, cumulative and current-semester HSGPA), school attendance, and behavioral (i.e., disciplinary) infractions as 
assessed at the conclusion of the 2013-2014 school year. 

ICCs were estimated for all outcome variables to assess the extent of school-level variance observed in the student 
outcome data. Significant amounts of clustering were observed by schools, with ICCs for the outcome variables estimated 
as follows: standardized test scores = .21, cumulative HSGPA = .13, current (Spring 2014) HSGPA = .11, attendance = .29, 
and behavioral infractions = .42. Given the extent of clustering present, we analyzed relationships between HSSN scores 
using both standard Pearson product-moment correlations and multilevel modeling (Raudenbush & Bryk, 2002). The 
latter method isolated variation within schools (between students) from variance existing between school units, allowing 
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us to evaluate the degree to which HSSN scores accounted specifically for between-student variations in outcomes (i.e., 
differences between students distilled of between-school variance). The amount of between-student variance accounted 
for the equal proportional reduction in the student-level variance component in a series of conditional two-level random 
coefficient models. The conditional student-level variance component was compared to its value in an unconditional 
model of variance in the same outcome. A separate conditional model was estimated for each outcome by HSSN dimension 
combinations (i.e., 14 models per outcome variable). All unconditional models specified a given outcome as the dependent 
variable, with no fixed effects and school-level random intercepts. Conditional models incorporated a given group-mean- 
centered HSSN score as the only covariate. 

Expected Attainment 

The literature in higher education has long recognized a connection between students’ own educational aspirations and 
their future attainment (Hanson, 1994; Messersmith & Schulenberg, 2008; Morgan, 2004; Trusty, 2000). In light of such 
findings, we explored relationships between HSSN scores and student anticipated levels of eventual educational attain¬ 
ment. Evidence for the validity of constructs as assessed by HSSN was garnered to the extent that noncognitive character¬ 
istics directly tied to college goals or academic behaviors were significantly associated with students’ expected attainment. 
Notably, there was only trivial between-school variance in the background questionnaire item related to anticipated attain¬ 
ment (ICC = .02). Responses to this variable were distributed relatively normally (skew = — .96, kurtosis = .13) across nine 
levels ranging from grade school and primary school or less to graduate or professional degree. 

Fairness 

Differential Item Functioning 

In validating the use of any assessment, it is important to examine item response data for differential item functioning 
(DIF). Where ordered categorical response data are concerned, DIF analyses assess whether there were significant dif¬ 
ferences in item response distributions across subgroups of interest, controlling for the overall score level in each group. 
DIF is a statistical phenomenon that occurs when there is “a difference in item performance between two comparable 
groups of examinees” (Dorans & Holland, 1992, p. 3). The word “comparable” is key here, differentiating DIF analyses 
from a situation in which subgroups of students might respond differently from one another on a scale due to an actual 
difference in their expressed levels of a given trait or ability. 

In the current study, DIF was examined among demographic subgroups. We considered both student gender and 
race/ethnicity categories (Black, White, Hispanic), where the latter comprised students identifying as a member of only 
one racial/ethnic group. Subgroup membership was self-reported by students, with institutionally reported data used in 
the minority of cases where they were available. Item response distributions were compared between two groups in a series 
of three analyses (male vs. female. Black vs. White, Hispanic vs. White) applied to each HSSN scale, where male or White 
served as the reference group. The group of students identifying as multiracial was not included in these analyses due to 
its heterogeneity. Interpreting the results of a DIF analysis as applying to a particular student subgroup would be unjus¬ 
tified in the case of multiracial students because the term and category will naturally shift in meaning and composition 
(respectively) with each new group of respondents. 

Students in our DIF analyses were matched across groups based on their mean score for each HSSN dimension. Criteria 
applied for assessing the level of DIF present for each polytomous HSSN item had three levels based on values of the 
Maentel-Haentszel ordinal procedure: AA (negligible/nonsignificant), BB (slight/weak), and CC (large/strong; N. L. Allen, 
Donoghue, & Schoeps, 2001; Dorans & Holland, 1992; Dorans & Kulick, 2006; Zwick, 2012). Negative DIF values indicated 
that the item was less strongly endorsed in the focal group versus the reference group, whereas positive values indicated the 
item was more strongly endorsed in the focal group. Following standard ETS procedures, for purposes of overall review, 
each HSSN item was assigned the lowest letter grade (i.e., greatest amount) of DIF exhibited across all three demographic 
comparisons (Dorans & Holland, 1992). Following both ETS and NAEP guidelines, the content of any items that exhibited 
CC DIF was examined by a panel of subject area and test construction experts to detect any construct-irrelevant factors 
that might have led to the observed disparity (and thus would have provided a rationale for removing the item from 
further use in HSSN; N. L. Allen et al., 2001). 
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Table 6 Domain and Subskill Crosswalk Between SuccessNavigator Assessments 


SuccessNavigator 


High School SuccessNavigator 


Academic Skills 

• Meeting Class Expectations 

• Organization 

Academic Skills 

Commitment 

• Commitment to College Goals 

• Institutional Commitment 

Achievement Motivation 

Self-Management 

• Academic Self-Efficacy 

• Sensitivity to Stress 

• Test Anxiety 

Self-Management 

Social Support 

• Connectedness 

Social Support 


• Institutional Support 

• Barriers to Success 

Lacking Barriers 


• Meeting Class Expectations 

• Organization 

• Perseverance 

• Commitment to College Degree Goals 

• Academic Self-Efficacy 

• Managing Test Anxiety 

• Managing Stress 

• Connectedness 

• School-Related Help Seeking 


Table 7 HSSN Measurement Model Fit Statistics 



^items 

x 1 

df 

CFI 

RMSEA 

90% Cl 

Overall sample {N = 3,187) 

Academic Skills 

28 

2,047.21 

347 

.933 

.039 

[.038, .041] 

Achievement Motivation 

19 

1,553.70 

151 

.955 

.054 

[.052, .056] 

Self-Management 

21 

1,040.39 

188 

.963 

.038 

[.035, .040] 

Social Support 

19 

832.96 

151 

.939 

.038 

[.035, .040] 

Lacking Barriers 

8 

398.36 

20 

.958 

.077 

[.071, .084] 

Index subsample (N = 1,593) 

Academic Skills 

28 

1,274.72 

347 

.941 

.041 

[.039, .043] 

Achievement Motivation 

19 

886.32 

151 

.963 

.055 

[.052, .059] 

Self-Management 

21 

679.74 

188 

.965 

.041 

[.037, .044] 

Social Support 

19 

542.33 

151 

.961 

.040 

[.037, .044] 

Lacking Barriers 

8 

238.69 

20 

.960 

.083 

[.074, .092] 

Reserve subsample (N = 1,594) 

Academic Skills 

28 

1,422.74 

347 

.928 

.044 

[.042, .046] 

Achievement Motivation 

19 

965.58 

151 

.955 

.058 

[.055, .062] 

Self-Management 

21 

671.19 

188 

.961 

.040 

[.037, .043] 

Social Support 

19 

564.27 

151 

.944 

.041 

[.038, .045] 

Lacking Barriers 

8 

318.32 

20 

.949 

.097 

[.088, .106] 


Note. CFI = confirmatory fit index; RMSEA = root mean square error of approximation. 


Results 

Dimensionality 
Measurement Models 

Owing to HSSN representing the adaptation of an existing assessment to a new population, our initial rubric for fitting 
correlated-traits models to the field trial data reflected the theoretical groupings of college subskills in the SuccessNavigator 
assessment into domains. Table 6 displays a crosswalk between the final HSSN subskill grouping and the construct orga¬ 
nization used by college SuccessNavigator (Markle et al., 2013). Although we used the college SuccessNavigator grouping 
as a starting point for determining the measurement structure of HSSN, it could only be followed approximately because 
HSSN both omits item content targeting institutional commitment and contains a significant proportion of new item 
content. Table 7 displays model fit statistics for CFA models fit to the index and reserve subsamples as well as the overall 
HSSN sample. Tables 8-12 present item parameter and R 2 (i.e., communality) estimates for each of these measurement 
models fit to the overall HSSN sample. 

As can be seen in Table 7, all models fit the data acceptably well across both independent subsamples and the overall 
dataset. Across all models, 95 items were retained from the original set of 100 candidate HSSN items. Although close in 
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Table 8 Academic Skills Domain — Measurement Model Factor Loadings and Item R 2 



Meeting Class Expectations 


Organization 


Perseverance 


Item 

Loading R 2 

Item 

Loading 

R 2 

Item Loading 

R 2 

Ml 

.73 .54 

Ol 

.59 

.35 

PI .73 

.53 

M2 

.58 .34 

02 

.74 

.54 

P2 .73 

.53 

M3 

.66 .43 

03 

.80 

.64 

P3 .47 

.22 

M4 

.71 .51 

04 

.65 

.42 

P4 .51 

.26 

M5 

.64 .41 

06 

.67 

.45 

P5 .56 

.31 

M9 

.52 .27 

07 

.70 

.49 

P6 .68 

.46 

M6 

.52 .27 

08 

.69 

.47 

P7 .82 

.67 

M7 

.74 .54 

09 

.66 

.44 

P9 .71 

.51 

M8 

.62 .38 




P10 .52 

.27 

05 

.78 .61 




Pll .67 

.45 

Note. Latent interfactor correlations in this model were as follows: Meeting Class Expectations was correlated .74 and .89 with Organi- 

zation and Perseverance, respectively. Organization correlated .61 with perseverance. 




Table 9 Achievement Motivation Domain 

— Measurement Model Factor Loadings and Item R 2 





Commitment to College Degree Goals 



Academic Self-Efficacy 


Item 

Loading 

R 2 


Item 

Loading 

R 2 


G1 

.78 

.61 


Al 

.87 

.76 


G2 

.89 

.78 


A2 

.85 

.73 


G3 

.82 

.68 


A3 

.52 

.27 


G4 

.85 

.71 


A8 

.64 

.41 


G5 

.78 

.62 


A4 

.49 

.24 


G6 

.84 

.71 


A9 

.71 

.51 


G7 

.83 

.68 


A7 

.46 

.21 


G8 

.67 

.45 


A5 

.71 

.51 


G9 

.72 

.52 


A6 

.66 

.43 


G10 

.89 

.80 






Note. Latent factors for Commitment to College Degree Goals and Academic Self-Efficacy were correlated .84. 



Table 10 

Self-Management Domain — Measurement Model Factor Loadings and Item R 2 





Managing Stress 



Managing Test Anxiety 



Item 

Loading 

R 2 

Item 


Loading 

R 2 


St 

.76 

.57 

T1 


.82 

.67 


S2 

.51 

.26 

T2 


.59 

.35 


S3 

.61 

.37 

T3 


.76 

.58 


S4 

.71 

.50 

T4 


.66 

.44 


S5 

.62 

.39 

T5 


.41 

.17 


S9 

.53 

.28 

T10 


.73 

.53 


S6 

.75 

.56 

T6 


.68 

.47 


S10 

.67 

.45 

T7 


.74 

.54 


S7 

.71 

.50 

T9 


.80 

.11 


S8 

.34 

.12 

Til 


.51 

.65 





T8 


.33 

.26 



Note. Latent factors for Managing Stress and Managing Test Anxiety were correlated .72. 


all cases, our final models fit to the overall sample did not always meet or exceed both typically held cutoff values for 
acceptable fit as indicated by the CFI and RMSEA (.95 and .06, respectively). As noted, these cutoffs were only considered 
approximate guidelines for acceptable fit. Given the practical application and scoring procedures intended for HSSN, we 
prioritized model parsimony over post hoc modifications (e.g., correlated uniqueness terms) that may have edged model 
fit beyond the preceding thresholds. Of course, the exhibition of overall model fit is a necessary but not sufficient condition 
for retaining a model in practice. Several other aspects of the results are presented in Tables 8-12. 
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Table 11 Social Support Domain — Measurement Model Factor Loadings and Item R 2 



Connectedness 



School-Related Help Seeking 


Item 

Loading 

R 2 

Item 

Loading 

R 2 

Cl 

.73 

.54 

N1 

.67 

.45 

C2 

.73 

.53 

N2 

.68 

.47 

C3 

.44 

.20 

N3 

.67 

.45 

C4 

.46 

.22 

N8 

.68 

.46 

C5 

.45 

.20 

N4 

.69 

.48 

C6 

.77 

.59 

N5 

.60 

.35 

C7 

.78 

.60 

N9 

.44 

.19 

C8 

.77 

.59 

N6 

.66 

.44 

C9 

.73 

.53 

N7 

.71 

.51 




N10 

.29 

.09 

Note. Latent factors for Connectedness and School-Related Help Seeking were correlated .63. 


Table 12 Lacking Barriers — Measurement Model Factor Loadings and Item R 2 



Item 



Loading 


R 2 

LI 



.69 


.48 

L2 



.76 


.58 

L7 



.66 


.43 

L3 



.51 


.26 

L4 



.39 


.15 

L5 



.38 


.14 

L8 



.76 


.57 

L9 



.78 


.61 


Within the Academic Skills model (Table 8), two original items were iteratively removed from further consideration 
after demonstrating significantly weaker relationships versus peer items to their hypothesized latent parent factor. On aver¬ 
age, the Academic Skills model accounted for 44% of the variance in latent variables presumed to underlie HSSN item 
responses. Although two of the latent factors in our Academic Skills model were highly correlated (Meeting Class Expec¬ 
tations and Organization, r = .89), we retained a three-factor model for two reasons. First, although the two constructs are 
thematically related, there is a distinct conceptual difference between meeting specific classroom expectations and being 
generally organized in one’s life and work. Second, we tested alternative models and found that a three-factor model 
demonstrated a significantly closer fit to the data than either a two-factor (A/ 2 = 492.71(2), p < .001) or unidimensional 
(A/ 2 = 2,114.59(3), p < .001) model encompassing the same HSSN items. 

All items in the Achievement Motivation domain model (Table 9) demonstrated adequate loadings on their respective 
parent factors, with the model accounting for 56% of latent item response variance on average. Although the two latent 
factors in this model were highly correlated (r= .84), a two-factor model was retained primarily due to the theoretical 
distinction between general academic self-efficacy and a specific commitment to obtaining a college degree. Moreover, 
the two-factor model exhibited a significantly closer fit to the data than an alternative unidimensional arrangement 
(A/ 2 = 216.72(1),p < .001). 

The situation regarding both the Self-Management (Table 10) and Social Support (Table 11) domain models was sim¬ 
ilar to that of the Achievement Motivation model. All items in the Self-Management model demonstrated adequate or 
strong factor loadings, whereas one item was removed from the Social Support model for demonstrating a weak standard¬ 
ized loading (.09). The Self-Management and Social Support models accounted for an average of 42% and 41% of latent 
item response variance, respectively. In both models, two factors were retained despite moderate interfactor correlations 
(r= .72 for Self-Management, r = .63 for Social Support). The rationale was consistent with the models discussed previ¬ 
ously in that within each model, both dimensions were theoretically distinct. For example, considering Self-Management, 
the management of stressful experiences in general is related but nonetheless distinct from dealing specifically with issues 
of test anxiety. A two-factor model also exhibited a significantly closer fit to the data than a unidimensional model in both 
the Self-Management (A/ 2 = 665.63(1), p < .001) and Social Support {A/ 2 = 349.09(1), p < .001) domains. 
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Table 13 Reliability and Descriptive Statistics for High School SuccessNavigator Domains and Subskills 



# Items 

“o 

M 

SD 

Skew 

Kurt 

Academic Skills 

28 

.94 

2.75 

.49 

.08 

-.34 

Meeting Class Expectations 

10 

.88 

2.95 

.56 

-.16 

-.48 

Organization 

8 

.86 

2.24 

.64 

.58 

-.09 

Perseverance 

10 

.87 

2.95 

.53 

-.13 

-.54 

Achievement Motivation 

19 

.95 

3.17 

.59 

-.75 

-.15 

Commitment to College Degree Goals 

10 

.95 

3.24 

.69 

-1.01 

.28 

Academic Self-Efficacy 

9 

.87 

3.08 

.58 

-.43 

-.48 

Self-Management 

21 

.91 

2.74 

.53 

-.31 

-.21 

Managing Test Anxiety 

10 

.88 

2.78 

.60 

-.30 

-.34 

Managing Stress 

11 

.86 

2.69 

.58 

-.33 

-.19 

Social Support 

19 

.90 

2.68 

.53 

.07 

-.32 

Connectedness 

9 

.87 

2.58 

.64 

.04 

-.45 

School-Related Help Seeking 

10 

.85 

2.77 

.57 

-.06 

-.48 

Lacking Barriers 

8 

.83 

3.25 

.54 

-.86 

.59 


Note. N = 3,187; a 0 = ordinal alpha; Skew = skewness; Kurt = kurtosis. 


Responses to items concerned with any external structural challenges experienced by students were modeled apart from 
the four general HSSN domains. A unidimensional model of Lacking Barriers (Table 12) fit to eight of the 10 items orig¬ 
inally hypothesized to assess such challenges both fit the data acceptably well and explained 40% of latent item response 
variance on average. Two of the original Lacking Barriers items were dropped in initial analyses for demonstrating rela¬ 
tively weak relationships with the latent construct. 


Reliability and Score Distributions 

Reliability and descriptive statistics for all HSSN scales are reported in Table 13. All exhibited acceptably high levels of 
reliability for a multidimensional assessment, with a 0 > .85 for all except Lacking Barriers (where a 0 was also acceptable 
at .83). As expected, due to their greater numbers of items, the general domains were observed to exhibit similar or higher 
levels of reliability than were the subskills. Scores were distributed approximately normally as evidenced by the lack of 
marked skewness or kurtosis, with absolute values of those statistics across all scales <1.01. 

Essential to note with regard to individual scores is that their point estimates would only be used for research purposes. 
In operation, scores referenced to the overall sample of HSSN field trial participants would be reported in a similar fashion 
to their format in the college version of the SuccessNavigator assessment. Score categories would be reported rather than 
point estimates to encourage school staff to avoid making fine distinctions between students based on individual HSSN 
scores. These categories would be assigned qualitative labels (e.g., developing, on target, strong) to encourage their inter¬ 
pretation as developmental guideposts within a larger student profile as opposed to exact determinations of a student’s 
likelihood of demonstrating behaviors indicative of each targeted construct. 

Validity Evidence 

Convergent and Discriminant Relationships 

Correlations between HSSN general domain scores are presented in Table 14, with correlations between specific skill scores 
presented in Table 15. These tables provide a sense of how strongly scores on each HSSN dimension relate to one another. 
Relationships between general domain scores were moderate on average 6 (M= .33, SD= ,29) 7 and followed expected 
patterns. For example, the two most highly correlated domains (Academic Skills and Achievement Motivation, r = .72) 
were also the two most theoretically similar in terms of their direct connection to behaviors and attitudes necessary for 
success in an academic environment. Of note in particular in Table 14, with regard to evidence for discriminant validity, 
are the relationships between Self-Management and its peer domains. Here we observed near-zero or relatively weak 
associations, demonstrating the distinctiveness of the constructs making up the Self-Management domain with respect to 
the other domains. The exception to this pattern was the moderate association observed between Self-Management and 
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Table 14 Correlations Among High School SuccessNavigator Domain Scores 



1 


2 



3 



4 

1. Academic Skills 

2. Achievement Motivation 

.72 









3. Self-Management 

.02+ 


.04 







4. Social Support 

.58 


.57 



.13 




5. Lacking Barriers 

.17 


.24 



.40 



.19 

Note. N = 3,187. 

p < .01 except where marked fp > .05. 










Table 15 Correlations Among High School SuccessNavigator Subskill Scores 








1 

2 

3 

4 

5 

6 

7 

8 

9 

Academic skills 










Meeting Class Expectations 

Organization 

.59 









Perseverance 

Achievement motivation 

.73 

.49 








Commitment to College Degree Goals 

.62 

.42 

.54 







Academic Self-Efficacy 

Self-Management 

.75 

-.02+ 

.42 

.72 

.70 






Managing Test Anxiety 

-.18 

.10 

-.12 

.15 





Managing Stress 

Social Support 

.09 

-.08 

.21 

-.02+ 

.22 

.62 




Connectedness 

.46 

.37 

.47 

.42 

.48 

.03+ 

.23 



School-Related Help Seeking 

.51 

.36 

.46 

.46 

.49 

.03+ 

.16 

.53 


Lacking Barriers 

.26 

-.02+ 

.20 

.19 

.28 

.32 

.41 

.08 

.25 


Note. N = 3,187. Bolded values indicate relationships between skills within the same general domain. 
p < .001 except where marked t p > -05. 


Lacking Barriers (r= .40), expected because a natural conceptual convergence exists between these areas to the extent 
that a student’s skill at managing stressful situations is connected to their ability to navigate specific challenges such as 
structural barriers to success. Lacking Barriers demonstrated weaker but statistically significant relationships with the 
other three domains. 

Table 15 presents similar evidence for the convergent and discriminant validity of HSSN from the perspective of its 
specific skill scores. Here we again observed moderate relationships on average 8 (M = .34, SD= .29), with most scales 
following patterns as expected. An efficient way to summarize Table 15 in terms of the extent to which the results provide 
evidence for the convergent and discriminant validity of HSSN scores is to describe the pattern of relationships between 
scores contributing to the same general domain versus those contributing to different domains. As an example, consider 
the skills comprising the Self-Management domain (Managing Stress and Managing Test Anxiety). The absolute value of 
this relationship (r = .62) was greater in magnitude than all the other 14 relationships observed between these two and their 
eight peer HSSN-specific skill scores. 9 Of particular note with regard to the subskills making up Self-Management was 
the negative association observed between Managing Test Anxiety and Commitment to College Degree Goals. Although 
relatively weak in magnitude (r = —.12), one might expect this relationship to have been in the opposite direction (i.e., 
better management of testing situations being related to more ambitious educational goals). One explanation for the 
observed relationship may be that some degree of anxiety surrounding testing situations is actually helpful or reflective 
of simply caring a great deal about one’s performance on exams. Seen in this light, students who are simply disengaged 
from their academic work would also be expected to care less than their peers about pursuing postsecondary education. 
In other words, for some students, experiencing more test anxiety (i.e., lower levels of Managing Test Anxiety) may be an 
indication of motivation to perform well to achieve future educational goals. 

The situation was the same for the Social Support domain as for Self-Management, where the within-domain corre¬ 
lation between Connectedness and School-Related Help Seeking (r= .53) was greater in magnitude than any between- 
domain correlation involving those two scores. Considering Achievement Motivation, the within-domain correlation 
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Table 16 Correlations Between High School SuccessNavigator Scale Scores and Big 5 Personality Factors 



Agree. 

(JV= 3,111) 

Consc. 

(N = 3,108) 

Emot. 

St. (N = 3,114) 

Extra. 

(N = 3,111) 

Intel. 

(N = 3,108) 

Academic Skills 

.40 

.67 

.16 

.17 

.32 

Meeting Class Expectations 

.42 

.60 

.16 

.14 

.30 

Organization 

.26 

.50 

-.01* 

.11 

.14 

Perseverance 

.34 

.63 

.26 

.20 

.38 

Achievement Motivation 

.39 

.50 

.17 

.18 

.39 

Commitment to College Degree Goals 

.36 

.42 

.08 

.14 

.31 

Academic Self-Efficacy 

.35 

.53 

.26 

.20 

.42 

Self-Management 

-.04 

.18 

.64 

.16 

.11 

Managing Test Anxiety 

-.05 

.12 

.47 

.08 

.14 

Managing Stress 

-.02 1 ' 

.20 

.70 

.22 

.06 

Social Support 

.41 

.39 

.29 

.40 

.24 

Connectedness 

.38 

.30 

.31 

.50 

.23 

School-Related Help Seeking 

.33 

.38 

.20 

.20 

.20 

Lacking Barriers 

.13 

.26 

.31 

.01+ 

.05 


Note. Agree. = Agreeableness; Consc. = Conscientiousness; Emot. St. = Emotional Stability; Extra. = Extraversion; Intel. = Intellectual 
Orientation. 

p < .01 except where marked f p > -05. 


between Commitment to College Degree Goals and Academic Self-Efficacy (r= .70) was greater in magnitude than 12 
(86%) of the 14 between-domain correlations involving those two scores. Turning to Academic Skills, the average correla¬ 
tion between the three skills in this domain was .62. This value exceeded 15 (83%) of the 18 between-domain correlations 
including Meeting Class Expectations, Organization, and Perseverance. In the eight of nine cases where they were statis¬ 
tically significant, relationships between Lacking Barriers and the HSSN subskills were positive and moderate in strength. 
The extent to which students had avoided family or other structural challenges to their academic success was positively 
correlated with their reported psychosocial skill levels. 

Associations between HSSN scores and separately assessed Big 5 personality traits provide further evidence of the 
convergent and discriminant validity of the assessment. These associations are displayed in Table 16. Reliability of the 50- 
item IPIP assessment was acceptable in the overall sample for all five personality traits, with a 0 = .83 for Agreeableness, 
a Q = .78 for Conscientiousness, a D = .88 for Emotional Stability, a Q = .87 for Extraversion, and a 0 = .81 for Intellectual 
Orientation. As discussed, the constructs targeted by HSSN were each hypothesized to relate most strongly to one of three 
personality factors: Conscientiousness, Emotional Stability, and Extraversion. These hypotheses were generally supported 
by the pattern of relationships observed in the data. For example, Academic Skills and Achievement Motivation were 
more strongly related to Conscientiousness (r = .67 and r = .50, respectively) than to any other personality trait. Similarly, 
Self-Management was more strongly related to Emotional Stability (r = .64) than to any other personality trait. Social 
Support demonstrated approximately equivalent associations with Agreeableness (r= .41), Conscientiousness (r= .39), 
and Extraversion (r = .40). These patterns also held true with regard to the more specific subskills targeted by HSSN. All 
seven subskill scores outside Social Support were observed to correlate more highly with the same personality trait related 
most strongly to their parent general domain than they did with any of the other Big 5 traits. Looking in more detail at the 
subskills making up the Social Support domain, the observed patterns explain the domain’s less sharp distinctions between 
the Big 5 versus the other HSSN domains. To wit, Connectedness was most strongly related to Extraversion (r = .50) 
whereas School-Related Help Seeking (i.e., taking it upon one’s self to find supportive resources) was most strongly related 
to Conscientiousness (r= .38). 

Relationships With Student Outcomes Criteria 

Table 17 reports relationships between HSSN scores and student outcome variables. Nonparenthetical entries are Pearson 
pro duct-moment correlations. Parenthetical entries indicate the percentage of variation in student outcomes between 
children within schools accounted for by their respective HSSN scores. Values equal the proportional reduction in the 
student-level variance component as estimated via multilevel modeling, where each two-level random coefficients model 
included a given HSSN score as the only covariate. 
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Table 17 Relationships Between High School SuccessNavigator Scores and Student Outcomes 



Standardized 

test scores 
{N= 1,421) 

Cumulative 
HSGPA 
(N= 3,003) 

Current 
HSGPA 
(N= 1,907) 

Behavioral 
infractions 
( N= 1,092) 

Attendance 

rate 

(N= 2,533) 

% explainable variance 2 

78.8 


84.2 


85.5 


58.3 


71.1 


Academic Skills 

.15 

(1.5) 

.40 

(16.2) 

.43 

(18.0) 

-.16 

(3.2) 

.10 

(0.8) 

Meeting Class Expectations 

.19 

(2.6) 

.45 

(20.6) 

.49 

(23.6) 

-.18 

(3.7) 

.14 

(1.8) 

Organization 

.08 

(0.1) 

.27 

(7.5) 

.28 

(7.6) 

-.12 

(1.1) 

.04 

(0.0) 

Perseverance 

.12 

(1.3) 

.30 

(9.5) 

.33 

(10.6) 

-.12 

(2.3) 

.06 

(0.4) 

Achievement Motivation 

.26 

(5.8) 

.46 

(21.9) 

.46 

(22.3) 

-.15 

(3.7) 

.12 

(1.5) 

Commitment to College Degree Goals 

.18 

(2.8) 

.38 

(15.2) 

.37 

(15.1) 

-.12 

(2.8) 

.11 

(0.9) 

Academic Self-Efficacy 

.31 

(8.4) 

.48 

(24.7) 

.50 

(25.7) 

-.17 

(3.9) 

.12 

(1.8) 

Self-Management 

.13 

(1.8) 

.00+ 

(0.0) 

.01+ 

(0.0) 

-.03+ 

(0.0) 

.03+ 

(0.1) 

Managing Test Anxiety 

.16 

(3.1) 

-.01+ 

(0.0) 

.02+ 

(0.0) 

.00+ 

(-0.1) 

.01+ 

(0.0) 

Managing Stress 

.07 

(0.3) 

.01+ 

(0.0) 

.00+ 

(0.0) 

-.05+ 

(0.3) 

.04 

(0.1) 

Social Support 

.10 

(0.3) 

.22 

(4.4) 

.24 

(4.5) 

-.06 

(0.5) 

.07 

(0.4) 

Connectedness 

.15 

(0.6) 

.23 

(4.1) 

.21 

(3.0) 

-.07 

(0.3) 

.08 

(0.4) 

School-Related Help Seeking 

.02+ 

(0.0) 

.16 

(2.3) 

.21 

(4.0) 

-.03+ 

(0.4) 

.04 

(0.2) 

Lacking Barriers 

.03+ 

(0.2) 

.13 

(2.6) 

.14 

(2.8) 

-.05+ 

(1.1) 

.14 

(2.2) 


Note. HSGPA = high school grade point average. Nonparenthetical entries are Pearson product-moment correlations. All values are 
significant at p < .05 except where marked tp > -05. Parenthetical entries indicate the percentage of variation in Student Outcomes 
between children within schools accounted for by the respective HSSN scale scores. 

“Total % of potentially explainable variance between children within schools. Values equal (1 - intraclass correlation)*100, where the 
intraclass correlation was estimated via multilevel modeling. Each two-level, unconditional means model applied random intercepts 
for schools. 


Summarizing the entire table, it is clear that, in all cases where results were statistically significant, all relationships 
observed were in the expected direction. Higher levels of targeted HSSN constructs were associated with higher levels of 
academic achievement as well as increased attendance rates. Similarly, as expected, higher HSSN scores were generally 
associated with fewer instances of behavioral infractions. 

Turning to specific outcome relationships and their relative strengths, HSSN scores were associated more strongly on 
average with current semester (M = .27, SD= .19) and cumulative (M = .25, SD= .18) HSGPA than with standardized test 
scores (M = .14, SD = .08), behavioral infractions (M = —.09, SD = .06), or attendance rates (M = .08, SD = .04). In terms 
of criterion relationships offering evidence for the validity of HSSN, it is notable that the strongest associations with aca¬ 
demic outcomes were observed between HSSN general domain and subskill scores directly targeting academic as opposed 
to more general behavioral characteristics. For example, considering the specific subskills. Academic Self-Efficacy exhib¬ 
ited the highest associations with both cumulative and current semester HSGPA (r= .48 and r= .50, respectively) as 
well as with standardized test scores (r= .31). This was followed closely by Meeting Class Expectations, which showed 
similarly strong relationships with cumulative and current HSGPA (r = .45 and r = .49, respectively) and demonstrated 
the next highest correlation with test scores (r= .19) after Academic Self-Efficacy. These results contributed to HSSN 
domain scores following the same pattern, with markedly stronger relationships observed between academic outcomes 
and both the Achievement Motivation and Academic Skills domains than with either Self-Management or Social Support. 
Lastly, of note with regard to academic outcomes was that Self-Management (in particular its Managing Test Anxiety 
component) was positively associated with better student performance on standardized tests but exhibited no relationship 
to course grades (i.e., HSGPA), which are earned over a lengthy period of time in (typically) lower-stake contexts. 

Turning briefly to notable relationships with institutionally reported behavioral outcomes, these also followed expected 
patterns. For example, higher levels of Meeting Class Expectations and Academic Self-Efficacy were most strongly associ¬ 
ated with lower counts of behavioral infractions (r = —.18 and r= —.17, respectively). Also worth highlighting is that, 
although moderate in magnitude and tied with Meeting Class Expectations, a lack of reported structural barriers to 
academic success showed the greatest association with better attendance rates (r = .14) across all HSSN constructs. The 
similarly moderate associations between Lacking Barriers and academic achievement outcomes provide support for the 
validity of student reports of such challenges. 
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Switching focus to the extent of between-student within-school variance explained by HSSN scores, the paren¬ 
thetical values in Table 17 demonstrate similar patterns of relative magnitude as described previously concerning 
correlational estimates. In general, it is interesting to note the substantially lower proportion of variation observed 
between students within schools in behavioral versus academic outcomes. Although scores on each outcome vari¬ 
able were concorded to a common metric across schools (i.e., placed on the same numeric scale), this phenomenon 
may be attributable in part to greater natural variation in the standards used to code behavioral infractions and 
attendance between schools in comparison to markers of academic achievement and externally scored standardized 
tests. 

While, on average, HSSN scores accounted for the greatest amounts of variance between students in cumulative 
(M = 9.2%, SD = 8.8%) and current semester HSGPA (M = 9.8%, SD = 9.4%) versus standardized test scores and behav¬ 
ioral outcomes, it is useful to highlight a few values in particular given the large disparity in these estimates across 
constructs. For example, scores on Academic Self-Efficacy accounted for the largest proportion of variation between stu¬ 
dents with regard to four of the five outcomes measured (current semester HSGPA = 25.7%, cumulative HSGPA = 24.7%, 
standardized test scores = 8.4%, and behavioral infractions = 3.9%). Several other scales also accounted for double-digit 
percentages in student course grade outcomes (e.g., general Academic Skills and Achievement Motivation; subskills 
assessing Commitment to College Degree Goals, Perseverance, and Meeting Class Expectations). These findings were 
in sharp contrast to the Self-Management domain and its subskills, which showed poor explanatory power in terms of 
accounting for between-student variation in outcomes. Notably, the extent to which students lacked barriers to their 
academic success explained a greater proportion (albeit small in magnitude; 2.2%) of between-student variance than any 
other HSSN construct score. 

Expected Attainment 

N = 2,626 students (82% of the overall sample) responded to the relevant background survey item asking them to report 
their ultimate level of expected educational attainment. Seventy-six percent of these (N = 1,997) reported expecting to 
eventually attain a bachelor’s degree or above, while an additional 4.6% and 8.1% (i.e., 88.7% of the total responding 
sample) expected to earn an associate’s degree or at least some college or university credit, respectively. A further 2.7% 
of students expected to complete a vocational or technical school program, bringing the total proportion of responding 
students expecting to pursue education beyond their high school diploma to 91.4%. 

Associations between student levels of anticipated educational attainment and conceptually related HSSN scores pro¬ 
vided further evidence for the validity of the instrument’s content, with Commitment to College Degree goals (r= .45) 
and general Achievement Motivation ( r = .44) exhibiting the strongest positive relationships (both p<.01). The next 
strongest relationships with expected attainment were observed with Academic Self-Efficacy (r = .34), Meeting Class 
Expectations (r = .28), general Academic Skills (r = .27), and Perseverance (r = .24; allp < .01). Moderate and statistically 
significant (p < .01) associations were also observed between expected educational attainment and several other HSSN 
scores including general Social Support (r= .20), its subskills Connectedness (r = .19) and School-Related Help Seeking 
(r= .14), Organization (r= .18), and Lacking Barriers (r= .08). 


Differential Item Functioning 

Results of all DIF analyses are displayed in Table 18, which reports the number of items falling within each DIF cate¬ 
gory by HSSN construct. The maximum numbers of items flagged as exhibiting CC DIF in a given scale across the three 
demographic contrasts conducted were two for Academic Skills and Achievement Motivation in the male versus female 
comparison, three for Academic Skills in the Black versus White comparison, and five for Achievement Motivation in 
the Hispanic versus White comparison. Turning to observed CC DIF among the items making up each HSSN subskill, 
it is notable that the subskills making up each domain together tended to exhibit fewer CC DIF items than the overall 
domain itself. Taking the male versus female comparison of items making up the Academic Skills domain as an example, 
two items were flagged as exhibiting CC DIF at the domain level, but none did so at the subskill level. Inclusive of analyses 
where no CC DIF was observed at all, this type of finding held true across eight of the 12 domain-level DIF analyses 
conducted. 
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Table 18 Summary of Differential Item Functioning Categorizations between Demographic Subgroups Across High School Success- 
Navigator Scores 


Scale 

# Items 


Male/female 



White/Black 



White/Hisp 

anic 


CC+ 

BB+ 

AA 

BB- 

CC- 

CC+ 

BB+ 

AA 

BB- 

CC- 

CC+ 

BB+ 

AA 

BB- 

CC- 

Academic Skills 

28 

1 

3 

17 

6 

1 

0 

4 

19 

2 

3 

2 

0 

24 

2 

0 

Meeting Class Expectations 

10 

0 

1 

9 

0 

0 

1 

1 

7 

1 

0 

1 

0 

9 

0 

0 

Organization 

8 

0 

1 

7 

0 

0 

0 

0 

6 

2 

0 

0 

1 

6 

1 

0 

Perseverance 

10 

0 

1 

9 

0 

0 

0 

1 

8 

1 

0 

0 

0 

10 

0 

0 

Achievement Motivation 

19 

0 

0 

17 

0 

2 

0 

3 

14 

2 

0 

2 

2 

10 

2 

3 

Commitment to College Degree Goals 

10 

0 

0 

9 

1 

0 

0 

1 

9 

0 

0 

0 

0 

9 

1 

0 

Academic Self-Efficacy 

9 

0 

2 

5 

2 

0 

0 

1 

6 

2 

0 

0 

0 

8 

1 

0 

Self-Management 

21 

1 

1 

18 

1 

0 

0 

3 

14 

4 

0 

2 

0 

14 

4 

1 

Managing Test Anxiety 

11 

1 

1 

9 

0 

0 

0 

1 

8 

2 

0 

1 

1 

8 

1 

0 

Managing Stress 

10 

0 

0 

9 

1 

0 

2 

0 

6 

2 

0 

1 

0 

7 

1 

1 

Social Support 

19 

0 

0 

19 

0 

0 

0 

0 

17 

1 

1 

0 

1 

16 

1 

1 

Connectedness 

9 

0 

0 

9 

0 

0 

0 

1 

6 

1 

1 

0 

0 

8 

1 

0 

School-Related Help Seeking 

10 

0 

0 

10 

0 

0 

0 

0 

10 

0 

0 

0 

0 

10 

0 

0 

Lacking Barriers 

8 

0 

0 

8 

0 

0 

0 

1 

6 

1 

0 

0 

0 

7 

0 

1 


The content of all CC DIF items was resubmitted to the ETS fairness and sensitivity review panel for a second round of 
review, with none found to exhibit any potentially problematic elements. Separately, these items as well as their associated 
DIF results were analyzed by a demographically diverse panel of experts in both test construction and noncognitive skills 
assessment. Following an in-depth review of all item content, the panel unanimously concluded that none of the flagged 
FfSSN items was susceptible to any obvious construct-irrelevant influences that might be expected to have led to the 
observed DIF. Thus, all items initially submitted for DIF analyses were retained for future use in HSSN. 

Discussion 

Summary 

In this report, we have provided detailed information regarding the underlying rationale, theoretical background, and 
intended uses of HSSN as an assessment of noncognitive attitudes and behaviors among high school students preparing 
for a transition to postsecondary education. HSSN was designed to serve as a developmental aid to both students and 
their school counselors or college advisors. With student support staff overburdened in many U.S. public schools and 
students themselves potentially uninformed about the skills and behaviors they will need to demonstrate (and cultivate) 
success in postsecondary education, HSSN is intended to provide a standardized rubric for gauging and facilitating student 
psychosocial development. 

We have further provided psychometric and other statistical evidence supporting the measurement structure of the 
assessment as well as the reliability and validity of scores representing each general and specific dimension. Evidence of 
statistically significant convergent, discriminant, and criterion-related relationships involving HSSN scores aligned with 
expected patterns. These relationships were demonstrated both internally among HSSN scores and between them and 
external measures of concurrent Big 5 personality traits, near-term academic and behavioral outcomes, and students’ 
anticipated future levels of educational attainment. Finally, we examined the fairness of HSSN item content through a 
multifaceted analysis of DIF. Results showed that relatively few of the assessment’s items demonstrated significant DIF 
across three different demographic subgroup comparisons, with none found to be culturally insensitive or biased in their 
content after review by an expert panel. 

Future Directions 

Perhaps the most important direction for future research is to attempt to gather college enrollment and/or performance 
data for the current sample (e.g., admission, enrollment status, institution type, collegiate GPA). Students participating in 
the HSSN field trial study were still enrolled in high school during this report’s production. Compiling college enrollment 
data will be vital to generating predictive evidence for the validity of HSSN scores to the extent that they portend divergent 
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postsecondary outcomes. It may also be of interest to use enrollment data to develop a predictive index capable of gauging 
a current high school student’s likelihood of successfully transitioning to higher education. 

One clear limitation of the current study is its focus on students enrolled in traditional public and charter high school 
environments as opposed to other populations of precollege students. Future studies to expand the potential applicability 
of HSSN might investigate its adaptation to adult learners in pursuit of a high school equivalency credential (e.g., the 
HiSET® test, GED) or those following other alternative pathways to postsecondary education. 

As a newly available assessment targeting noncognitive attitudes and behaviors, HSSN provides an opportunity to 
pursue multiple lines of future research (beyond those discussed previously) with significant potential value to the field. 
Chief among these may be studies deploying the assessment longitudinally in an effort to understand the stability of its 
measurement characteristics over time as well as its sensitivity to developmental change. A similar line of research would 
see HSSN employed as an interim or outcome assessment within the context of an intervention study (e.g., to gauge the 
impact of a program designed to improve noncognitive skills levels). 

There are also several other applied studies that would be interesting to conduct using HSSN data along with student 
outcomes. As an example, latent profile analysis is expected to prove a useful strategy for developing typologies of late 
high school noncognitive skill expression. Such analyses should in turn enable the targeted delivery of student support 
services customized to the specific needs of empirically identified student subgroups (Olivera-Aguilar, Markle, & Robbins, 
2014; Pastor, Barron, Miller, & Davis, 2007). Further studies making use of multilevel modeling would also provide a 
more nuanced picture than is currently available of the extent to which both school- and student-level noncognitive 
characteristics (in combination with other contextual factors) contribute to the prediction of student academic success in 
high school and beyond. 

Notes 

1 Though McDonough (2006) used 2003 data from the Common Core of Data (CCD) and U.S. Department of Education, the 
student to counselor ratio of 478:1 is still representative of current circumstances in national public schools. Using 2012-2013 
data from the same CCD source, this ratio increased only slightly to 482:1. 2012-2013 data were retrieved from the State 
Nonfiscal Public Elementary/Secondary Education Survey through the National Center for Education Statistics 
(http://nces.ed.gov/ccd/stnfis.asp). Including only the United States and District of Columbia, the following variables from the 
CCD dataset and calculation were used: MEMBER (total students) / TOTGUI (Total guidance counselors/directors). 

2 https://www.schoolcounselor.org/asca/media/asca/home/RoleStatement.pdf 

3 E.g., NACAC (http://www.nacacnet.org/), National College Advising Corps (http://advisingcorps.org/), Gear Up 
(http://www2.ed.gov/programs/gearup/index.html). 

4 Adjustments were made automatically using the TYPE=COMPLEX option in M plus. 

5 Because mean item scores are a relatively simple representation of student levels on HSSN constructs versus a model-based 
scoring approach such as maximum a posteriori (MAP), we conducted analyses to confirm the adequate comparability of our 
estimated scores to MAP factor scores. These analyses showed that mean item scores performed essentially identically to MAP 
scores in our data. We estimated MAP scores directly from our item-level CFA models using M plus. Each HSSN scale s MAP 
score was then correlated with its corresponding mean item score. Across the 10 HSSN subskills, the minimum correlation 
between MAP and mean item scores was r = .95. This confirmed that the two scoring methods would produce variables expected 
to function essentially identically in subsequent analyses. 

6 Relationships involving Lacking Barriers were not included in this and other similar calculations. 

7 All average correlations reported in this study were calculated by converting raw r values to z statistics using Fisher s 
transformation, with average z values then back-transformed to the reported average r values (Corey, Dunlap, & Burke, 1998). 

8 Relationships involving Lacking Barriers were not included in this and other similar calculations. 

9 This and the other similar comparisons made in this paragraph are qualitative in nature. Our intention is to provide a simple 
descriptive summary of the pattern of results without engaging in statistical comparisons of all possible combinations of 
estimated values. 
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